Search for: All records

Abstract Existing models for evaluating early-stage designs typically assume that solutions follow dominant solving approaches in the problem domain. While effective for comparing alternatives within dominant solving approaches, these models often undervalue or overlook solutions that use atypical or novel approaches, especially when these differ significantly in key design variables. This bias prematurely constrains the design space and becomes a pressing problem as firms increasingly leverage nontraditional sources of innovation and creativity (e.g., through crowdsourcing). To address this, we introduce a modeling approach that enables comparison of design solutions from multiple solving paradigms. It represents engineering design as a problem-solving process, with solutions generated by selecting concepts and embodiments to achieve specific functions. The model simulates how different solvers navigate this process based on their expertise, producing a variety of solutions rather than those limited to dominant strategies. The quality of each solution is represented as a probability distribution over performance and cost. The model’s effectiveness is demonstrated using a robotic arm design problem, leveraging a dataset from a large-scale field experiment. Results show that the model can estimate performance and cost across different solving approaches, capturing valuable solutions that traditional models would miss. This is particularly significant when evaluating designs from nontraditional solvers, as they are more likely to diverge from dominant solving paradigms. As firms increasingly turn to nontraditional sources of expertise for innovation, this modeling approach could enable comprehensive identification and fair assessment of a range of design solutions. 

Abstract Engineering design relies heavily on heuristics, yet there is a lack of systematic methods for identifying and validating design heuristics. This paper introduces a computational approach to representing engineering design problems that involve decomposition and assignment decisions, facilitating systematic extraction of generalizable heuristics. We model design processes using a Markov Decision Process (MDP) framework, characterizing problems through attributes of the problem space, solver capabilities, and trade-offs embedded within preference functions. Reinforcement learning methods are employed to learn optimal policies, from which we extract inclusionary and exclusionary heuristics using Gaussian Mixture Models. The effectiveness of the approach is demonstrated through two case studies: solver-aware system architecting (SASA) for a robotic arm design and sequential information acquisition in parametric design optimization. The results highlight the context-dependent nature of learned heuristics, demonstrating how problem complexity, designer preferences, and solver characteristics influence their selection. 

Abstract The engineering design research community has developed numerous modeling approaches for evaluating the goodness of early-stage designs. However, these models often assume that solving approaches dominant in the problem domain will be pursued. Although these models are effective in terms of evaluating alternative solutions within that dominant solving approach, they cannot fairly compare solutions that may use alternative or novel concepts. Consequently, the value of solutions that use atypical solving approaches is either disregarded or underestimated. This is a pressing problem that prematurely constrains the design space, as atypical solving approaches are likely to be increasingly utilized as firms increasingly look to non-traditional sources of creativity and innovation (e.g., through crowdsourcing). To address these limitations, this paper introduces a modeling framework that is capable of generating and comparing solutions from multiple solving paradigms on a common basis. The model represents engineering design as a problem-solving process where solutions are characterized by different concepts and embodiments used to realize various functionalities. The model captures how different solvers would navigate this process to generate design solutions by simulating the likelihood that the solvers will select particular principles and embodiments based on their expertise. This, in return, yields a diverse set of solutions derived from different solving approaches - instead of a homogeneous set that is only composed of dominant solutions. The model also outputs goodness of design solutions as a probability distribution over performance and associated costs. The value of the modeling framework is demonstrated using the reference problem of designing a robotic arm, leveraging a dataset previously collected from a large-scale field experiment. The results indicate that the model can estimate the performance and cost of solutions using different solving approaches, not just the dominant approach. Findings suggest that this capability is more important when non-traditional solvers are involved in the design process, as a larger fraction of their solutions would have been missed by models considering only the dominant solving approach. Hence, we anticipate that as firms increasingly turn to non-traditional sources of expertise for innovation, this type of modeling approach could enable comprehensive identification and fair assessment of a variety of design solutions. 

Abstract Systems design involves decomposing a system into interconnected subsystems and allocating resources to teams responsible for designing each subsystem. The outcomes of the process depend on how well limited resources are allocated to different teams, and the strategy each team uses to design the subsystems. This article presents an approach based on hierarchical reinforcement learning (RL) to generate heuristics for solving complex design problems under resource constraints. The approach consists of formulating systems design problems as hierarchical multiarmed bandit (MAB) problems, where decisions are made at both the system level (allocating budget across subsystems) and the subsystem level (selecting heuristics for sequential information acquisition). The approach is demonstrated using an illustrative example of a race car optimization in The Open Racing Car Simulator (TORCS) environment. The results indicate that the RL agent can learn to allocate resources strategically, prioritize the subsystems with the greatest influence on overall performance, and identify effective information acquisition heuristics for each subsystem. For example, the RL agent learned to allocate a larger portion of the budget to the gearbox subsystem, which has a higher-dimensional design space compared to other subsystems. The results also indicate that the extracted heuristics lead to convergence to high-performing car configurations with greater efficiency when compared to using Bayesian optimization for design. 

Abstract The crowdsourcing literature has shown that domain experts are not always the best solvers for complex system design problems. Under certain conditions, novices and specialists in adjacent domains can provide novel solutions at lower costs. Additionally, the best types of solvers for different problems are dependent on the architecture of complex systems. The joint consideration of solver assignment and system decomposition, referred to as solver-aware system architecting (SASA), expands traditional system architecting practices by considering solver characteristics and contractual incentive mechanisms in the design process and aims to improve complex system design and innovation by leveraging the strengths of domain experts, crowds, and specialists for different parts of the problem. The joint consideration of problem decomposition and solver assignment decisions in SASA renders the design space exponentially more complex. Therefore, new computationally efficient and mathematically rigorous methods are needed to explore this high-dimensional space and extract reliable heuristics. To address this need, this paper presents a computational approach using a Markov decision process (MDP) formulation, Q-learning, and Gaussian mixture models. Together, these techniques explore the large space of possible solver–module assignments by modeling the sequential nature of solver assignment decisions, capturing these temporal dependencies, thereby enabling optimization for long-term expected rewards, and analyzing reward distributions. The approach identifies heuristics for solver assignment based on the designer’s preference for cost-performance trade-off through the parameterized reward function. The approach is demonstrated using a simple and idealized golf problem, which has characteristics similar to design problems, including how the problem is decomposed into interdependent modules and can be solved by different solvers with different strengths that interact with the module type. The results show that the proposed approach effectively elicits a rich set of heuristics applicable in various contexts for the golf problem and can be extended to more complex systems design problems. 

Abstract Existing literature on information sharing in contests has established that sharing contest-specific information influences contestant behaviors, and thereby, the outcomes of a contest. However, in the context of engineering design contests, there is a gap in knowledge about how contest-specific information such as competitors’ historical performance influences designers’ actions and the resulting design outcomes. To address this gap, the objective of this study is to quantify the influence of information about competitors’ past performance on designers’ belief about the outcomes of a contest, which influences their design decisions, and the resulting design outcomes. We focus on a single-stage design competition where an objective figure of merit is available to the contestants for assessing the performance of their design. Our approach includes (i) developing a behavioral model of sequential decision making that accounts for information about competitors’ historical performance and (ii) using the model in conjunction with a human-subject experiment where participants make design decisions given controlled strong or weak performance records of past competitors. Our results indicate that participants spend greater efforts when they know that the contest history reflects that past competitors had a strong performance record than when it reflects a weak performance record. Moreover, we quantify cognitive underpinnings of such informational influence via our model parameters. Based on the parametric inferences about participants’ cognition, we suggest that contest designers are better off not providing historical performance records if past contest outcomes do not match their expectations setup for a given design contest. 

Abstract Extracting an individual's scientific knowledge is essential for improving educational assessment and understanding cognitive tasks in engineering activities such as reasoning and decision making. However, knowledge extraction is an almost impossible endeavor if the domain of knowledge and the available observational data are unrestricted. The objective of this paper is to quantify individuals' theory-based causal knowledge from their responses to given questions. Our approach uses directed acyclic graphs (DAGs) to represent causal knowledge for a given theory and a graph-based logistic model that maps individuals' question-specific subgraphs to question responses. We follow a hierarchical Bayesian approach to estimate individuals' DAGs from observations.The method is illustrated using 205 engineering students' responses to questions on fatigue analysis in mechanical parts. In our results, we demonstrate how the developed methodology provides estimates of population-level DAG and DAGs for individual students. This dual representation is essential for remediation since it allows us to identify parts of a theory that a population or individual struggles with and parts they have already mastered. An addendum of the method is that it enables predictions about individuals' responses to new questions based on the inferred individual-specific DAGs. The latter has implications for the descriptive modeling of human problem-solving, a critical ingredient in sociotechnical systems modeling. 

Abstract Heuristics are essential for addressing the complexities of engineering design processes. The goodness of heuristics is context-dependent. Appropriately tailored heuristics can enable designers to find good solutions efficiently, and inappropriate heuristics can result in cognitive biases and inferior design outcomes. While there have been several efforts at understanding which heuristics are used by designers, there is a lack of normative understanding about when different heuristics are suitable. Towards addressing this gap, this paper presents a reinforcement learning-based approach to evaluate the goodness of heuristics for three sub-problems commonly faced by designers: (1) learning the map between the design space and the performance space, (2) acquiring sequential information, and (3) stopping the information acquisition process. Using a multi-armed bandit formulation and simulation studies, we learn the suitable heuristics for these individual sub-problems under different resource constraints and problem complexities. Additionally, we learn the optimal heuristics for the combined problem (i.e., the one composing all three sub-problems), and we compare them to ones learned at the sub-problem level. The results of our simulation study indicate that the proposed reinforcement learning-based approach can be effective for determining the quality of heuristics for different problems, and how the effectiveness of the heuristics changes as a function of the designer’s preference (e.g., performance versus cost), the complexity of the problem, and the resources available.